# Elasticsearch monitoring

SkyWalking leverages elasticsearch-exporter for collecting metrics data from Elasticsearch. It leverages OpenTelemetry
Collector to transfer the metrics to
[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).

## Data flow

1. The elasticsearch-exporter collect metrics data from Elasticsearch.
2. OpenTelemetry Collector fetches metrics from elasticsearch-exporter via Prometheus Receiver and pushes metrics to
   SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to
   filter/calculate/aggregate and store the results.

## Setup

1. Setup [elasticsearch-exporter](https://github.com/prometheus-community/elasticsearch_exporter).
2. Set up [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/#kubernetes). The example for OpenTelemetry Collector configuration, refer
   to [here](../../../../test/e2e-v2/cases/elasticsearch/otel-collector-config.yaml).
3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).

## Elasticsearch Monitoring

Elasticsearch monitoring provides multidimensional metrics monitoring of Elasticsearch clusters as `Layer: ELASTICSEARCH` `Service` in the OAP. In each cluster, the nodes are represented as `Instance` and indices are `Endpoint`s.

### Elasticsearch Cluster Supported Metrics

| Monitoring Panel          | Metric Name                                                 | Description                                                                                     | Data Source            |
|---------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------------------------------|------------------------|
| Cluster Health            | meter_elasticsearch_cluster_health_status                   | Whether all primary and replica shards are allocated                                            | elasticsearch-exporter |
| Tripped Of Breakers       | meter_elasticsearch_cluster_breakers_tripped                | Tripped for breaker                                                                             | elasticsearch-exporter |
| Nodes                     | meter_elasticsearch_cluster_nodes                           | Number of nodes in the cluster.                                                                 | elasticsearch-exporter |
| Data Nodes                | meter_elasticsearch_cluster_data_nodes                      | Number of data nodes in the cluster                                                             | elasticsearch-exporter |
| Pending Tasks             | meter_elasticsearch_cluster_pending_tasks_total             | Cluster level changes which have not yet been executed                                          | elasticsearch-exporter |
| CPU Usage Avg. (%)        | meter_elasticsearch_cluster_cpu_usage_avg                   | Cluster level percent CPU used by process                                                       | elasticsearch-exporter |
| JVM Memory Used Avg. (%)  | meter_elasticsearch_cluster_jvm_memory_used_avg             | Cluster level percent JVM memory used                                                           | elasticsearch-exporter |
| Open Files                | meter_elasticsearch_cluster_open_file_count                 | Open file descriptors                                                                           | elasticsearch-exporter |
| Active Primary Shards     | meter_elasticsearch_cluster_primary_shards_total            | The number of primary shards in your cluster. This is an aggregate total across all indices     | elasticsearch-exporter |
| Active Shards             | meter_elasticsearch_cluster_shards_total                    | Aggregate total of all shards across all indices, which includes replica shards                 | elasticsearch-exporter |
| Initializing Shards       | meter_elasticsearch_cluster_initializing_shards_total       | Count of shards that are being freshly created                                                  | elasticsearch-exporter |
| Delayed Unassigned Shards | meter_elasticsearch_cluster_delayed_unassigned_shards_total | Shards delayed to reduce reallocation overhead                                                  | elasticsearch-exporter |
| Relocating Shards         | meter_elasticsearch_cluster_relocating_shards_total         | The number of shards that are currently moving from one node to another node                    | elasticsearch-exporter |
| Unassigned Shards         | meter_elasticsearch_cluster_unassigned_shards_total         | The number of shards that exist in the cluster state, but cannot be found in the cluster itself | elasticsearch-exporter |

### Elasticsearch Node Supported Metrics

| Monitoring Panel                | Unit      | Metric Name                                                                                                                                          | Description                                        | Data Source            |
|---------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------|------------------------|
| Node Rules                      |           | meter_elasticsearch_node_rules                                                                                                                       | Node roles                                         | elasticsearch-exporter |
| JVM Memory Used                 | MB        | meter_elasticsearch_node_jvm_memory_used                                                                                                             | Node level JVM memory used size                    | elasticsearch-exporter |
| CPU Percent                     | %         | meter_elasticsearch_node_process_cpu_percent                                                                                                         | Node level percent CPU used by process             | elasticsearch-exporter |
| Documents                       |           | meter_elasticsearch_node_indices_docs                                                                                                                | Count of index documents on this node              | elasticsearch-exporter |
| Segments                        |           | meter_elasticsearch_node_segment_count                                                                                                               | Count of index segments on this node               | elasticsearch-exporter |
| Disk Free Space                 | GB        | meter_elasticsearch_node_all_disk_free_space                                                                                                         | Available space on all block device                | elasticsearch-exporter |
| Open Files                      |           | meter_elasticsearch_node_open_file_count                                                                                                             | Open file descriptors                              | elasticsearch-exporter |
| Process CPU Usage Percent       | %         | meter_elasticsearch_node_process_cpu_percent                                                                                                         | Percent CPU used by process                        | elasticsearch-exporter |
| OS CPU usage percent            | %         | meter_elasticsearch_node_os_cpu_percent                                                                                                              | Percent CPU used by the OS                         | elasticsearch-exporter |
| Load Average                    |           | meter_elasticsearch_node_os_load1 <br/>meter_elasticsearch_node_os_load5<br/>meter_elasticsearch_node_os_load15                                      | Shortterm, Midterm, Longterm load average          | elasticsearch-exporter |
| JVM Memory Usage                | MB        | meter_elasticsearch_node_jvm_memory_nonheap_used<br />meter_elasticsearch_node_jvm_memory_heap_used<br/>meter_elasticsearch_node_jvm_memory_heap_max | JVM memory currently usage by area                 | elasticsearch-exporter |
| JVM Pool Peak Used              | MB        | meter_elasticsearch_node_jvm_memory_pool_peak_used                                                                                                   | JVM memory currently used by pool                  | elasticsearch-exporter |
| GC Count                        |           | meter_elasticsearch_node_jvm_gc_count                                                                                                                | Count of JVM GC runs                               | elasticsearch-exporter |
| GC Time                         | ms/min    | meter_elasticsearch_node_jvm_gc_time                                                                                                                 | GC run time                                        | elasticsearch-exporter |
| All Operations ReqRate          |           | meter_elasticsearch_node_indices_*_req_rate                                                                                                          | All Operations ReqRate on node                     | elasticsearch-exporter |
| Indexing Rate                   | reqps     | meter_elasticsearch_node_indices_indexing_index_total_req_rate<br />meter_elasticsearch_node_indices_indexing_index_total_proc_rate                  | Indexing rate on node                              | elasticsearch-exporter |
| Searching Rate                  | reqps     | meter_elasticsearch_node_indices_search_fetch_total_req_rate<br />meter_elasticsearch_node_indices_search_query_time_seconds_proc_rate               | Searching rate on node                             | elasticsearch-exporter |
| Total Translog Operations       |           | meter_elasticsearch_node_indices_translog_operations                                                                                                 | Total translog operations                          | elasticsearch-exporter |
| Total Translog Size             | MB        | meter_elasticsearch_node_indices_translog_size                                                                                                       | Total translog size                                | elasticsearch-exporter |
| Tripped For Breakers            |           | meter_elasticsearch_node_breakers_tripped                                                                                                            | Tripped for breaker                                | elasticsearch-exporter |
| Estimated Size Of Breaker       | MB        | meter_elasticsearch_node_breakers_estimated_size                                                                                                     | Estimated size of breaker                          | elasticsearch-exporter |
| Documents Count                 | KB/s      | meter_elasticsearch_node_indices_docs                                                                                                                | Count of documents on this node                    | elasticsearch-exporter |
| Merged Documents Count          | count/s   | meter_elasticsearch_node_indices_merges_docs_total                                                                                                   | Cumulative docs merged                             | elasticsearch-exporter |
| Deleted Documents Count         |           | meter_elasticsearch_node_indices_docs_deleted_total                                                                                                  | Count of deleted documents on this node            | elasticsearch-exporter |
| Documents Index Rate            | calls/s   | meter_elasticsearch_node_indices_indexing_index_total_req_rate                                                                                       | Total index calls per second                       | elasticsearch-exporter |
| Merged Documents Rate           | MB / s    | meter_elasticsearch_node_indices_merges_total_size_bytes_total                                                                                       | Total merge size per second                        | elasticsearch-exporter |
| Documents Deleted Rate          | docs/s    | meter_elasticsearch_node_indices_docs_deleted                                                                                                        | Count of deleted documents per second on this node | elasticsearch-exporter |
| Count Of Index Segments         |           | meter_elasticsearch_node_segment_count                                                                                                               | Count of index segments on this node               | elasticsearch-exporter |
| Current Memory Size Of Segments | MB        | meter_elasticsearch_node_segment_memory                                                                                                              | Current memory size of segments                    | elasticsearch-exporter |
| Network                         | bytes/sec | meter_elasticsearch_node_network_send_bytes<br/>meter_elasticsearch_node_network_receive_bytes                                                       | Total number of bytes sent and receive             | elasticsearch-exporter |
| Disk Usage Percent              | %         | meter_elasticsearch_node_disk_usage_percent                                                                                                          | Used space on block device                         | elasticsearch-exporter |
| Disk Usage                      | GB        | meter_elasticsearch_node_disk_usage                                                                                                                  | Used space size of block device                    | elasticsearch-exporter |
| Disk Read                       | KBs       | meter_elasticsearch_node_disk_io_read_bytes                                                                                                          | Total kilobytes read from disk                     | elasticsearch-exporter |
| Disk Write                      | KBs       | meter_elasticsearch_node_disk_io_write_bytes                                                                                                         | Total kilobytes write from disk                    | elasticsearch-exporter |

### Elasticsearch Index Supported Metrics

| Monitoring Panel                              | Unit  | Metric Name                                                                                                                                                                                                                                                    | Description                                                                            | Data Source            |
|-----------------------------------------------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|------------------------|
| Documents Primary                             |       | meter_elasticsearch_index_indices_docs_primary                                                                                                                                                                                                                 | Count of documents with only primary shards on all nodes                               | elasticsearch-exporter |
| Deleted Documents Primary                     |       | meter_elasticsearch_index_indices_deleted_docs_primary                                                                                                                                                                                                         | Count of deleted documents with only primary shards                                    | elasticsearch-exporter |
| Data Primary                                  | GB    | meter_elasticsearch_index_indices_store_size_bytes_primary                                                                                                                                                                                                     | Current total size of stored index data with only primary shards on all nodes          | elasticsearch-exporter |
| Data                                          | GB    | meter_elasticsearch_index_indices_store_size_bytes_total                                                                                                                                                                                                       | Current total size of stored index data with all shards on all nodes                   | elasticsearch-exporter |
| Segments Primary                              |       | meter_elasticsearch_index_indices_segment_count_primary                                                                                                                                                                                                        | Current number of segments with only primary shards on all nodes                       | elasticsearch-exporter |
| Segments Memory Primary                       | MB    | meter_elasticsearch_index_indices_segment_memory_bytes_primary                                                                                                                                                                                                 | Current size of segments with only primary shards on all nodes                         | elasticsearch-exporter |
| Segments                                      |       | meter_elasticsearch_index_indices_segment_count_total                                                                                                                                                                                                          | Current number of segments with all shards on all nodes                                | elasticsearch-exporter |
| Segments Memory                               | MB    | meter_elasticsearch_index_indices_segment_memory_bytes_total                                                                                                                                                                                                   | Current size of segments with all shards on all nodes                                  | elasticsearch-exporter |
| Indexing Rate                                 |       | meter_elasticsearch_index_stats_indexing_index_total_req_rate<br/>meter_elasticsearch_index_stats_indexing_index_total_proc_rate                                                                                                                               | Indexing rate on index                                                                 | elasticsearch-exporter |
| Searching Rate                                |       | meter_elasticsearch_index_stats_search_query_total_req_rate<br/>meter_elasticsearch_index_stats_search_query_total_proc_rate                                                                                                                                   | Searching rate on index                                                                | elasticsearch-exporter |
| All Operations ReqRate                        |       | meter_elasticsearch_index_stats_*_req_rate                                                                                                                                                                                                                     | All Operations ReqRate on index                                                        | elasticsearch-exporter |
| All Operations Runtime                        |       | meter_elasticsearch_index_stats_*_time_seconds_total                                                                                                                                                                                                           | All Operations Runtime/s on index                                                      | elasticsearch-exporter |
| Avg. Search Time Execute / Request            | s     | meter_elasticsearch_index_search_fetch_avg_time<br/>meter_elasticsearch_index_search_query_avg_time<br/>meter_elasticsearch_index_search_scroll_avg_time<br/>meter_elasticsearch_index_search_suggest_avg_time                                                 | Search Operation Avg. time on index                                                    | elasticsearch-exporter |
| Search Operations Rate                        | req/s | meter_elasticsearch_index_stats_search_query_total_req_rate<br/>meter_elasticsearch_index_stats_search_fetch_total_req_rate<br/>meter_elasticsearch_index_stats_search_scroll_total_req_rate<br/>meter_elasticsearch_index_stats_search_suggest_total_req_rate | Search Operations ReqRate on index                                                     | elasticsearch-exporter |
| Shards Documents                              |       | meter_elasticsearch_index_indices_shards_docs                                                                                                                                                                                                                  | Count of documents per shards on index                                                 | elasticsearch-exporter |
| Documents (Primary Shards)                    |       | meter_elasticsearch_index_indices_docs_primary                                                                                                                                                                                                                 | Count of documents with only primary shards on index                                   | elasticsearch-exporter |
| Documents Created Per Min (Primary Shards)    |       | meter_elasticsearch_index_indices_docs_primary_rate                                                                                                                                                                                                            | Documents rate with only primary shards on index                                       | elasticsearch-exporter |
| Total Size Of Index (Primary Shards)          | MB    | meter_elasticsearch_index_indices_store_size_bytes_primary                                                                                                                                                                                                     | Current total size of stored index data in bytes with only primary shards on all nodes | elasticsearch-exporter |
| Documents (All Shards)                        |       | meter_elasticsearch_index_indices_docs_total                                                                                                                                                                                                                   | Count of documents with all shards on index                                            | elasticsearch-exporter |
| Documents Created Per Min (All Shards)        |       | meter_elasticsearch_index_indices_docs_total_rate                                                                                                                                                                                                              | Documents rate with only all shards on index                                           | elasticsearch-exporter |
| Total Size Of Index (All Shards)              | MB    | meter_elasticsearch_index_indices_store_size_bytes_total                                                                                                                                                                                                       | Current total size of stored index data in bytes with all shards on all nodes          | elasticsearch-exporter |

## Customizations

You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found
in `/config/otel-rules/elasticsearch/elasticsearch-cluster.yaml, /config/otel-rules/elasticsearch/elasticsearch-node.yaml, /config/otel-rules/elasticsearch/elasticsearch-index.yaml`.
The Elasticsearch dashboard panel configurations are found in `/config/ui-initialized-templates/elasticsearch`.
